Referencing system

ABSTRACT

A reference imaging system including a planar reference piece. The reference imaging system further includes a three-axis gantry for positioning the planar reference piece at a plurality of points in a 3D coordinate system. Additionally, the reference imaging system includes a yaw actuator for adjusting the yaw angle of the object. Furthermore, the reference imaging system includes a pitch actuator for adjusting the pitch of the object. Moreover, the reference imaging system includes a computer processing unit for controlling the 3D position, pitch and yaw of the planar reference piece.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present U.S. patent application is a continuation and claims thebenefit of priority to U.S. patent application Ser. No. 16/705,649 filedDec. 6, 2019, which is a divisional and claims the benefit of priorityto U.S. patent application Ser. No. 16/038,161 filed Jul. 17, 2018,which is related to and claims the benefit of priority to U.S.provisional patent application 62/533,587 filed Jul. 17, 2017, thedisclosures of which are hereby incorporated by reference.

TECHNICAL FIELD

The present application relates to 3D object imaging, and morespecifically, to a referencing system for general imaging includinghyperspectral imaging, RGB imaging, greyscale imaging etc.

BACKGROUND

Image calibration with white referencing is one important step ofscientific imaging. The goal of the calibration is to eliminate theimpact from uneven lighting conditions. Typically, shortly before orafter the raw image of the target object is taken, a white tile isplaced at the same position as the object and is imaged too. For eachimage pixel, the pixel value in the raw image is divided by thecorresponding pixel value of the white reference image:

Corrected Image from the camera=Image of sample/Image of white tile

Sometimes the dark reading image is also taken by closing the apertureor simply putting on the lens cover, in which case:

Corrected Image from the camera=(Image of sample−Dark readingImage)/(Image of white tile−Dark reading Image)

The above method works well for flat objects, but when the object hascomplicated 3D shapes this method has serious problems because 1) theobject surface may be at different depth distances from the camera,where the lighting intensity can differ a lot from where the flat whitetile is located, and 2) the object surface may be at many differenttilted angles, which will severely change the reflectance not only inintensity, but also in color. Take plant leaf reflectance for example,where the PROSAIL model (http://teledetection.ipgp.jussieu.fr/prosail/)shows the different leaf angles completely change the reflectancespectra, which may cause 300% change in color index calculation such asNDVI, as shown in FIG. 1.

The problem may be solved by replacing the flat white tile with a 3Dwhite referencing. The 3D white reference should have exactly the samesize and 3D shape as the target object. Since each target object isdifferent, one solution can be achieved by placing a 3D scanner and a 3Dprinter on the spot. Every time a new object arrives, it is scanned bythe 3D scanner. The scanning result is then sent to the 3D printerimmediately to print out the 3D white reference. This 3D white referenceis then scanned for the white reference image, which is used tocalibrate the raw image of the object. Preliminary data from experimentsconfirm the improved calibration quality with 3D reference compared with2D flat reference. FIG. 2 shows the spectra of multiple points on a 3Dobject calibrated with 3D and flat references separately. The object ismade of uniform material and color, so the difference in spectra betweenthe different points can only be from lighting and angle variation. Asobserved from the figure, the 3D reference did a far better job than theflat reference. In addition, plant leaves were also imaged, rotated bydifferent angles. The result in FIG. 3 shows the similar improvedcalibration quality of 3D(sloped) reference compared with flatreference.

However, producing the 3D white reference for each object is anexpensive and impractical approach, incurring increased processing timeand resources. Therefore, improvements are needed in the field.

BRIEF DESCRIPTION OF THE DRAWINGS

One or more embodiments are illustrated by way of example, and not bylimitation, in the figures of the accompanying drawings, whereinelements having the same reference numeral designations represent likeelements throughout. It is emphasized that, in accordance with standardpractice in the industry, various features may not be drawn to scale andare used for illustration purposes only. In fact, the dimensions of thevarious features in the drawings may be arbitrarily increased or reducedfor clarity of discussion.

FIG. 1 illustrates effect of leaf angle of reflectance.

FIG. 2 illustrates a spectra of multiple points on a 3D objectcalibrated with 3D and flat references separately

FIG. 3 illustrates improved calibration quality of 3D(sloped) referencecompared with flat reference

FIG. 4 illustrates a referencing system in accordance with one or moreembodiments.

FIG. 5 illustrates a multi-axis moving apparatus in accordance with oneor more embodiments.

FIG. 6 illustrates a multi-axis moving apparatus in accordance with oneor more embodiments.

FIG. 7 illustrates a reference semi-sphere.

FIG. 8 illustrates a multi-axis moving apparatus in accordance with oneor more embodiments.

FIG. 9 illustrates a multi-axis moving apparatus in accordance with oneor more embodiments.

DETAILED DESCRIPTION

In the following description, some aspects will be described in termsthat would ordinarily be implemented as software programs. Those skilledin the art will readily recognize that the equivalent of such softwarecan also be constructed in hardware, firmware, or micro-code. Becausedata-manipulation algorithms and systems are well known, the presentdescription will be directed in particular to algorithms and systemsforming part of, or cooperating more directly with, systems and methodsdescribed herein. Other aspects of such algorithms and systems, andhardware or software for producing and otherwise processing the signalsinvolved therewith, not specifically shown or described herein, areselected from such systems, algorithms, components, and elements knownin the art. Given the systems and methods as described herein, softwarenot specifically shown, suggested, or described herein that is usefulfor implementation of any aspect is conventional and within the ordinaryskill in such arts.

The present disclosure provides a referencing system which calibratesimages. The referencing system includes a multi-axis moving apparatus(e.g. FIGS. 5, 6, and 8) that has a reference piece mounted. In FIG. 5,the reference piece has a dimension that is greater than a dimension ofa pixel. In FIG. 6, the reference piece has a dimension such that anentirety of a target object is contained within the length of thereference piece. In FIG. 8, the reference piece has a dimension suchthat an entirety of the target object is contained within a planarcross-section of the reference piece. The referencing system furtherincludes a camera and a 3D scanner. In some embodiments, the cameraincludes an RGB camera or a hyperspectral camera. In at least oneembodiment, the 3D scanner includes a 3D surface scanner, TOF camera,stereo camera, or a LiDAR.

In operation, step 1 begins with the referencing system collecting andstoring a library of images of the reference piece at a multitude ofpositions and slope angles using the camera and the 3D scanner. In atleast one embodiment, step 1 includes collecting a library of images ofall possible locations, orientations, and slopes of the reference piece.The library is stored in a processing unit such as a computer. In step2, a target object is scanned by the camera and the 3D scanner, therebyproducing a camera image and a 3D scanner image, respectively. Thetarget object can include a plant, a piece of meat, sculpture, or any 3Dobject. In step 3, each pixel from the camera image is matched with acorresponding pixel in the 3D scanner image so as to determine a 3Dlocation, orientation, and slope of this pixel in the camera image. Inone or more embodiments, step 3 is performed by the processing unit suchas a computer. This step results in determination of location,orientation, and slope of all pixels in camera image, according to atleast one embodiment.

In step 4, the determined 3D locations, orientations, and slopes ofvarious pixels in the camera image are used to identify relevantcorresponding images within the library. The relevant correspondingimages within the library have locations, orientations and slopes thatare commensurate with the determined 3D locations, orientations andslopes of various pixels of the camera image. In one or moreembodiments, step 4 is performed by the processing unit such as acomputer. In step 5, the relevant corresponding images are then used tovirtually reconstruct a 3D reference image of the target object. In oneor more embodiments, step 5 is performed by the processing unit such asa computer. In step 6, the camera image taken from the camera iscalibrated based on the reconstructed 3D reference image. In one or moreembodiments, the calibration includes using the 3D reference image toreference the camera image. In one or more embodiments, step 6 isperformed by the processing unit such as a computer.

The reference platform automatically generates the reference imagelibrary. FIG. 5 illustrates one embodiment having a multi-axis movementapparatus, upon which a reference piece is mounted as shown. Theplatform includes an x-axis gantry and motor, a y-axis gantry and motor,a z-axis gantry and motor, a yaw axis motor, and a pitch axis motor anda mounting base as illustrated. To build the image library, the whitereference piece is moved to each location and angle to be scanned by the3D scanner. The illustrated embodiment provides a 5 degrees of freedom(DOF) robotic arm holding a white reference. The 3 motor driven lineargantries move the white tile to each x-y-z position, while the pitch andyaw axis motors rotate the white piece to each combination of pitch andyaw angles. The movement and rotation are controlled by the computerprocessing unit and synchronized with the imager. In this way the systemautomatically takes an image of the white reference piece at eachposition and angle, thereby constructing and storing the library ofwhite reference images.

In one or more embodiments, the system of FIG. 5 first collects thewhite reference library data with proper spatial and angular resolutionsin the imaging space. This only needs to be done periodically (up toseveral months, for example) until the lighting, mechanical structure,and camera setting have noticeable changes. To collect the whitereference data, the system moves the tile to each of the coordinatepositions in the x,y,z system, and each yaw and pitch angles. At eachposition, the system records an image of the tile and the associatedreflectance data. Next, the 3D scanner is calibrated so the computerprocessing unit is able to match each original pixel with a 3D depthpixel. Next, the system uses the library to 3D reference the images inimaging arrays. The system records the image of the object and records a3D scan of the object. At each 3D pixel, besides reading the (x, y, z)information, the system also calculates the direction (pitch and yawangels) of the object surface at the point. This can be quickly done bya using a surface regression algorithm, for example. Then, for eachoriginal pixel, the system checks the x,y,z, pitch, yaw angels data withthe matching 3D pixel (by the algorithm above), and uses that data tolook up the white tile's reflection at the same position and directionfrom the library, and use that to reference the original value of thatpixel.

Once drawback of the embodiment shown in FIG. 5 is the long time ittakes to construct the entire library. For example, assuming a 1 cubicmeter imaging space, and 50 mm desired special resolution and 10 degreesangle resolution for the image library, it requires taking 20{circumflexover ( )}3*18{circumflex over ( )}2=2,592,000 images. With a normal pushbroom hyperspectral camera system, each scanning cycle takes about 10seconds, so the total scanning duration will be about 300 days nonstop.Of course, the imaging time may be reduced by using a lower resolution,but at the expense of image quality.

FIG. 6 shows a further embodiment of a white referencing system for pushbroom line scanning hyperspectral imaging systems. The illustratedembodiment reduces the 5 DOF to 3 DOF, thereby dramatically increasingspeed: The embodiment utilizes a white reference strip instead of awhite piece so the imager can take an image of multiple pixels at atime; the white strip will be placed both horizontally and vertically,and only a single rotating motor is needed for both yaw and pitchangles, depending on the direction of the white strip. In the sameexample above, the total scanning time becomes 20{circumflex over( )}2*18*2=14,400 images or 40 hours, which is 180 times faster than theembodiment of FIG. 5. Although 40 hours is still long, this procedure isonly needed once for every imaging period of 1-2 months for instance. Inoperation, the embodiment of FIG. 6 positions the white reference tilestrip to both horizontal and vertical directions. At each direction, thetile strip is moved to each of the x and z positions in the coordinatesystem, and rotates the tile strip to each of the angles for pitch andyaw. The image data and associated reflectance is then recorded. Theremainder of the process is the same as described with respect to theembodiment of FIG. 5.

FIGS. 7 and 8 show a further embodiment, which includes a plurality ofsemispherical white reference objects mounted on a panel as shown. Thegrid of semispherical reference objects are distributed on the panel,and each semispherical object provides references at a plurality ofdifferent angles. In this way, the original 5 DOF requirement is reducedto only 1 DOF (we only need to move the panel up and down at differentheights), thereby reducing the total processing time to a matter ofminutes. In operation, the embodiment of FIGS. 7 and 8 move the tile toeach of the z positions. At each z position, and image the panel andassociated reflectance data are recorded. The spatial resolution of thewhite reference library depends on the density of the sphere array. Inorder to increase this resolution, the array panel can slide in both xand y directions in certain embodiments to locate the spheres atdifferent positions and repeat this step again for several iterations.The remainder of the process is the same as described with respect tothe embodiment of FIG. 5.

The spatial resolution of the embodiment of FIGS. 7 and 8 is thedistance between 2 neighboring semi-sphere reference objects. If higherresolution is needed, 2 motors can be added to enable the movement in xand y directions as shown in FIG. 9. This will provide any spatialresolution needed, but it should not increase the scanning time too muchsince the maximum moving distance is just the distance between 2neighboring semispherical white reference objects. In variousembodiments, each of the semi-sphere reference objects are coated withpolytetrafluoroethylene, PVC, or a polymer. In one or more embodiments,each of the semi-sphere reference objects can have any other colorexcept for black. In some embodiments, each of the semi-sphere referenceobjects are white in color.

The invention is inclusive of combinations of the aspects describedherein. References to “a particular aspect” and the like refer tofeatures that are present in at least one aspect of the invention.Separate references to “an aspect” (or “embodiment”) or “particularaspects” or the like do not necessarily refer to the same aspect oraspects; however, such aspects are not mutually exclusive, unless soindicated or as are readily apparent to one of skill in the art. The useof singular or plural in referring to “method” or “methods” and the likeis not limiting. The word “or” is used in this disclosure in anon-exclusive sense, unless otherwise explicitly noted.

The invention has been described in detail with particular reference tocertain preferred aspects thereof, but it will be understood thatvariations, combinations, and modifications can be effected by a personof ordinary skill in the art within the spirit and scope of theinvention.

1. A reference imaging system, comprising: a planar reference piece; athree-axis gantry for positioning the planar reference piece; a yawactuator for adjusting the yaw angle of the object; a pitch actuator foradjusting the pitch of the object; and a computer processing unit forcontrolling the 3D position, pitch and yaw of the planar referencepiece.
 2. The system of claim 1, further comprising an imager forrecording images and reflectance data of the object.
 3. The system ofclaim 1, wherein the computer processing unit records the reflectancedata of the object at the plurality of points.
 4. The system of claim 3,wherein the computer processing unit compares the reflectance data ofthe planar reference piece with reflectance data of a sample todetermine a corrected image of the sample.
 5. A reference imagingsystem, comprising: a reference piece; a two-axis gantry for positioningthe reference piece; a yaw actuator for adjusting the yaw angle of anobject; a pitch actuator for adjusting the pitch of the object; and acomputer processing unit for controlling the 3D position, pitch and yawof the reference piece.
 6. The system of claim 5, further comprising animager for recording images and reflectance data of the object.
 7. Thesystem of claim 5, wherein the computer processing unit records thereflectance data of the object at the plurality of points.
 8. The systemof claim 7, wherein the computer processing unit compares thereflectance data of the reference piece with reflectance data of asample to determine a corrected image of the sample.
 9. The system ofclaim 1, wherein the planar reference piece comprises any color exceptfor black.
 10. The system of claim 5, wherein the reference piececomprises any color except for black.
 11. The system of claim 1, whereinthe planar reference piece has a dimension that is greater than adimension of a pixel.
 12. The system of claim 5, wherein the referencepiece has a dimension such that an entirety of the object is containedwithin the length of the reference piece.